On rationality of verbal subsets in a group 

A. Myasnikov, V. Roman'kov 
March 22, 2011 

Abstract 

Let Fbea free non-abelian group. We show that for any group word w 
the set w[F] of all values of w in F is rational in F if and only if w[F] = 1 
or w[F] — F. We generalize this to a wide class of free products of groups. 
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1 Introduction 

In this paper we study the structure and complexity of the verbal sets in free 
groups and free products of groups. The main result of the paper shows that 
proper verbal subsets of free non-abelian groups are not rational. We generalize 
this to a wide class of free products of groups and other groups which have such 
free products as their quotients. In particular, the result holds for arbitrary 
finitely generated non-abclian rcsidually free groups, pure braid groups, or non- 
abelian right angled Artin groups. 

Following Gilman [5] we define for a given group G the set Rat(G) of all 
rational subsets of G as the closure of the set of all finite subsets of G under the 
rational operations: union, product, and generation of a submonoid (Kleene's 
star operation). Sometimes, according to the standard practice, we refer to 
the rational subsets of a finitely generated free monoid as regular subsets. It is 
known (see [5]) that a subset L of a group G is rational in G if and only if L is 
accepted by a finite automaton over G (see definitions in Section [2]). 
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Let F(X) be a free group with a basis X and W C a subset of -F(X). 

An clement g in a group G is called a W -element if 3 is the image in G of some 
word ic e ^ under some homomorphism F(X) — > G. By W[G] we denote 
the set of all VF-elements in G. The set W[G] generates the verbal subgroup 
W(G). The verbal subgroups of groups were intensely studied in group theory 
especially with respect to relatively free groups and varieties of groups. We refer 
to the books [15] , [21] for the general facts in this area. 

W -width (or W -length) is one of the key notions concerning verbal subgroups 
W(G) of a group G. The W-length lw(g) of an clement g G W{G) is the the 
minimal natural number n such that g is a product of n W-elements in G or their 
inverses. The VF-width of the verbal subgroup W(G) is sup{lw{g) | 9 G W(g)}. 
It is usually assumed that the set W is finite. In this case the set W[G] is 
just a verbal set w(G) for a suitable single word w. Furthermore, for a finite 
W if the verbal subgroup W(G) is of finite W- width then it is equal to w[G] 
for some single word w (not necessary from W). In particular, the length of 
W(G) depends on the set W. From now on we will always assume that W 
is just a singleton W — {w} and refer to the related verbal set and the W- 
lcngth as w[G] and w-length. Usually, it is very hard to compute the wdength 
of a verbal subgroup w(G) or the u>-length of an element g G w(G). The 
first question of this type goes back to the Ore's paper [16] where he asked 
whether the commutator length (i.e., the [x, y]-length) of every element in a 
non-abelian finite simple group is equal to 1 (Ore Conjecture). Only recently 
the conjecture was established by Liebeck, O'Brian, Shalev and Tiep [12] . For 
recent spectacular results on the w-length in finite simple groups we refer to the 
papers [HI [22] and a book [21] . 

Sometimes, for example in the presence of negative curvature, it is convenient 
to replace the unruly standard VF-length with a more smooth stable Vy-length, 
which is defined for an element g G G as the limit lim„_ ) . 00 (which always 

exists). In some sense the stable commutator length relates to an L 1 filling norm 
with Q coefficients, introduced by Gromov in [9], see also Gersten's paper [7]. 
In [10] Gromov studied stable commutator length and its relation with bounded 
cohomology. We refer to a book [5] by Calegary on stable length of elements in 
groups. 

On the other hand, Calegari showed in [S] that if a group G satisfies a non- 
trivial law then the stable commutator length is equal to for every element 
from [G,G]. In particular, in solvable groups the stable commutator length is 
not very helpful, instead, the standard VF-length was studied intensely. We 
refer to papers [53], [TJ5], [2D], and a book [2TJ on the standard width of verbal 
subgroups in groups. 

Properties of verbal sets W[G] themselves play an important part in group 
theory. For example, the Membership Problem (MP) to the set W[G) in G is 
equivalent of solving in G the homogeneous equations of the type w(X) = g, 
where w G W and g G G. The Endomorphism Problem in G, which asks to 
decide for given elements g, h G G if there is an endomorphism G End(G) such 
that (j>(g) = h, is just a particular case of such a problem. Since the Diophantine 
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Problem (of solving arbitrary equations) is decidable in a free group F [Tl] , MP 
to the verbal sets in F is also decidable. However, Razborov showed in [17] that 
the solution sets of quadratic homogeneous equations of the type II" =1 [xi, y t ] = g 
may have a very complex structure. 

In this paper we focus on the complexity of verbal sets in free groups from the 
formal language theory view point. The goal is to determine the proper place 
of various verbal sets of a free group F (and some other "free- like" groups) in 
the hierarchy of formal languages. Since proper verbal subsets of F arc not 
rational in F, they are not regular in F, so they sit in some higher levels of the 
hierarchy of formal languages. What these levels are precisely is an interesting 
open problem. 

The paper is organized as follows. Section 2 contains necessary definitions 
and facts about rational subsets in groups. In Section 3 we discuss Rhemtulla's 
gap theorem , which is the main technical tool of our approach to verbal sets 
in free groups and free products of groups. In Section 4 we establish some key 
lemmas on general properties of rational verbal subsets, while in Section 5 we 
prove the main results of the paper. 

2 Preliminaries 

Let F = F(X) be a free group with basis X = {xi, . . . ,x n , . . .}, viewed as 
the set of reduced words in I U I' 1 with the standard multiplication. Fix 
w = w(x\,X2, ...,x n ) £ F(X). An clement g of a group G is called a w-element 
if g = w(gi,g2, ■■■,g n ) for some <?i, . . . ,g n £ G. We denote the set of all w- 
elements of G by w[G]. A subset MCGisa verbal subset of G if M = w[G] for 
some word w £ F(X). The subgroup w(G) generated by w[G] is the w-verbal 
subgroup of G, and a subgroup of G is called verbal if it is equal to w(G) for 
some w. 

A word w is said to be proper if there exist groups G and H such that 
w[G] ^ 1 and w[H] ^ H, in fact, in this case 1 ^ w[G x H] ^ G x H. 
Any element w = w(xi, . . . , x n ) £ F(X) can be written as a product 

t-i to t„ I 

w = x 1 x 2 ...x^w , 

where w' = w'(x\, . . . , x n ) £ [F,F]. Since the exponents t\,t2, ■■■,t n depend 
only on the element w the number e(w) = gcd(ti, t 2l t n ) is well-defined (here 
we put e(w) = if t\ = . . . = t n = 0). If e(w) = then we refer to w as 
a commutator word. A non-trivial commutator word is obviously proper. If 
e(w) > then there exist integers r%, f2, r n such that Y^i=i Ti ^ i = e ' so ^ or 
an arbitrary group G and an element g £ G one has w(g Tl , g T2 , g Tn ) = g e . 
In particular, e(w) = 1 implies that w[G] = G for every group G, so w is not 
proper. If e(w) > 1 then w is proper, which can be seen in an infinite cyclic 
group. In other words, a non-trivial word w is proper if and only if e(w) =/= 1. 

Let M be a monoid. For L C M by L* we denote the submonoid of M 
generated by L. The set Rat(M) C 2 A/ , of all rational subsets of M, is defined 
as the smallest (with respect to inclusion) subset which contains all finite subsets 
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of M and closed under the following operations (here Li,L 2 ,Z/ are subsets of 
M): 

• Union: (L\, £ 2 ) — > L\ U Z 2 - 

• Product: (L\, L 2 ) —> L\Li = {ab \ a G L\, b G L 2 }. 

• Submonoid generation: L — > L*. 

It follows from the definition above that every rational set L G Rat(M) in a 
monoid M can be presented in a form 

L = U>l =1 a il E* 1 ...a lU E* u a lM+1 (1) 

where all coefficients a,j are in M and each Eij is a rational subset of M. 

We define a complexity function c : Rat(M) -> N as follows. Put c(L) = 
if and only if L is finite. Suppose now that rational sets L e Rat(M) with 
c(L) < n — 1 arc defined. Then for a set L £ Rat(M) we put c(L) = n if and 
only if c(L) ^ n — 1, but either L = L\ U L 2 , or L = i 1 _L 2 , or I = L*, for some 
Li e Rat(M) with c(i<) < n - 1, i = 1, 2. 

It is easy to see that if G is a group then for any element g E G and a set 
i G Rat(G) one has 

cGT 1 ^) - c(L). (2) 
A finite AI —automaton is a tuple A = (Q,6,qo, F) where 

• Q is a finite set of states, 

• 9o G Q is the initial state, 

• f C Q is the set of terminating states, 

• SCQxMxQ is a finite relation, termed the transition relation. 

One can view an automaton A = (Q,5,qo,F) as a directed M-labellcd graph 
(possibly with multiple edges) , with the set of vertices Q and where two vertices 
it, v are connected by an edge u — > v labelled by m G M if and only if (u, m, v) G 
S. As usual, a path p in A from a vertex u to a vertex v is a sequence of edges 

(u , mi, wi), (iii,m 2 ,u 2 ), ■ • ■ , (u k -i,m k ,u k ) 

such that u = uq and u = The Za&e/ A(p) of p is the product mi . . . m k G AI . 
A successful path is a path from go to a vertex in F. 

The subset L(A) C M (the set of all elements in M accepted by A) is defined 

as 

L(A) = {X(p) p is a successful path in A}. 

It was shown in [5] that for any monoid M and any subset L C M the 
following equivalence holds: 



L G Rat(M) 



L = L(A) for some automaton A over M. 
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Gilman showed in [5] that any rational subset L of a group G generates in 
G a finitely generated (and so rational) subgroup. Hence if a verbal set w[G] is 
rational then the verbal subgroup w(G) is finitely generated. In the case when 
G is a free non-abelian group, or more generally, a non-trivial free product 
G = A*B, every normal subgroup of infinite index is not finitely generated (see 
[T3] and [3]), and so it is not rational. Hence the case when the verbal subgroup 
w(G) has finite index in G becomes the most interesting in our study. 

3 Free products and Rhemtulla's criterion 

Let G = A * B be a free product of non-trivial groups A and B. Each clement 
u ^ 1 of G can be uniquely written in its reduced form u — u\U2-~u m , where 
Ui G A U B \ {1} (i = 1, 2, m); and for every i the elements m, m+i are from 
different groups A and B. The number m is the syllable length of u, denoted by 
Put suppiu) = {u\, . . . , u m }. Furthermore, the reduced form of a non-trivial 
element ueG can be uniquely written as 

u = r^ 1 ...r^ 1 vi...v k r 1 ...r t , 

where either fc = 1, or k > 1 and v\Vk ^ 1. We refer to v\...Vk as the core of u 
and denote it by u. 

As usual one can define a cyclically reduced form u° of an element u € 
G. Namely, let the core u of u is given in the reduced form u = u\U2---u m . 
Then if u\,u m are from the different factors then u° = u. Otherwise, u° = 

(u 2 ■ ..«m_l, {u m Ui)). 

In [18] A.H. Rhcmtulla introduced a useful technique of gap functions for 
G = A* B. To explain, suppose one of the groups, say B, has an element b G B 
such that b ^ b^ 1 . Let u = UiU2---u m be a non-trivial element in G given in 
its reduced form. A subsequence . . . ,Ui + 2k of the reduced form of u 

is called a b—gap in u of length 2k — 1 if Ui = Ui + 2k = & and 7^ 6 for any 
i < j < i + 2k. 

For fc = 1, 2, . . . denote by Sb,k(u) the number of b— gaps of length 2k — 1 in 
it. For a positive integer e put 7^(14) to be the number of values k such that 
5b,k{u) ^ 6 b -i ik (u)(mode). 

Rhemtulla's criterion [18]. -Lei G = A * B and b £ B as above. Then for 
any word w(xi, . . . ,x n ) with e = e(w) > 1 the function 7^ is bounded on the 
set w[G]. 

4 Positive elements 

A sign function on a group G is a function p : G —> { — 1, 1} such that: 

• P(i) = i; 

• for any f,g G G if p(f) = l,p(g) = 1 then p(fg) = 1. 
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The set of positive elements Pos(G) = {g £ G | = 1} forms a submonoid 
in G. Conversely, a group G with a distinguished submonoid M C G admits a 
sign function p M such that Pm(9) = 1 if and only if g £ M. We refer to groups 
with sign functions as s-groups. Elements from M are called positive, all others 
- negative. The following examples are important in our context. 

Example 4.1. Let C = (a) be an infinite cyclic group generated by a. Then 
p(a n ) = 1, if n > 0, and p(a n ) = — 1 if n < is a sign function. 

Generalizing the example above, we get the following 

Example 4.2. Let G be a group with a generating set X. Then the submonoid 
mon(X) = X* generated by X gives a sign function px on G. 

Example 4.3. Let G = A * B be a free product of two non-trivial s— groups 
A, B. The function p : G — > { — 1, 1} such that p(g) — 1 if and only if all factors 
in the reduced form of g £ A * B are positive, is a sign function on G, termed 
the standard free product sign function. 

When applied to a free group F with basis X the examples above give the 
standard notion of a positive word in F . 

Notice, that there might be "zero divisors" in G relative to Pos{G), i.e., some 
elements x,y £ G, not both positive, such that xy £ Pos(G). For example, if 
u £ G is negative then uu^ 1 is positive. To separate the natural cases like 
uu~ x above (which is easy to deal with) from the harder ones we introduce the 
following notion. We say that a sign function p on G is reduced if it has the 
following property: 

• for any two subsets of elements S,T of G if ST C Pos(G) then here exists 
an element u £ G such that Su -1 C Pos(G) and uT C Pos(G). 

In particular, if uT C Pos(G) then there is an element uo such that uu^ 1 £ 
Pos(G) and u T C Pos(G). 

Notice also, that if p is reduced then for a fixed set S there is a single u that 
works for all subsets T as above; a similar claim holds for a fixed set T. 

We say that a sign function p on G is strongly reduced if it is reduced and 
has the following property: 

• any product of two negative elements is negative. 

Remark 4.4. The sign function on the infinite cyclic group from Example 14. II 
is strongly reduced. 

Lemma 4.5. The standard sign function (see Example \4-3\ ) on a free product 
G = A * B of two non-trivial s— groups with reduced sign functions is reduced. 
In particular, the standard sign function on a free group is reduced. 

Proof. Let S,T be non-empty subsets of G such that ST C Pos(G). For an 
element u £ S\Pos(G) written in its reduced form u = u\U2 . . .u%. . .Uh denote 
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by i = i(u) such an index that u, is negative, but all the factors u\, . . . , lij— l are 
positive. Similarly, for an element u = v\v 2 ...Vj...vi ET\ Pos{G) written in its 
reduced form let j = j(v) be the index such that Vj is negative but all the factors 
Vj + i, ... ,vi are positive. Notice, that the sets / = I(S) = {\u\ — i(u) + 1 | u G 
S \ Pos(G)} and J = J(T) = {j(v) |«gT\ Pos(G)} are bounded even if the 
sets S, T are infinite. Indeed, if, say, J is unbounded then for a given u € S there 
exists v' G T with \u\ < j(v'), in which case uv' is not positive, - contradicting 
the hypotheses of the lemma. Put max($) = 0, and ig = max (I), jo — max(J). 
The case S, T C Pos{G) when I = J = and z = j = is obvious. Assume 
that io < jo (the other case can be treated similarly). Let v G T\Pos(G) be an 
element with jo = j(v). Write v in the reduced form v = V\ . . . Vj -\Vj . . .vi, 
and assume that Vj lies, say, in B. Denote c = v± . . . Uj — i- Then for every 
element u G S the factor Vj — i cancels out in the reduced form of (otherwise 
a negative factor Vj occurs in the reduced form of uv). Hence the reduced form 
of each u e S is of the type u = U\ . . . w r ( u )C -1 for a suitable index r(u) and 
u r(u) G B. Consider now two cases. 

Case 1. io = jo- Let u G S \ Pos(G) be an element with io = i(u). 
In this case u = u\ . . . u^^c , where u r ui) G B is negative. The argument 
above shows that every element w G T can be written in the reduced form 
v = cvj ...v m , where Vj G B. It follows that for any u G S*, w G T one 
has u« = Ui . . . (u r ( u )Vj ) ■ ■ ■ v m , where all the factors, including {u r ( u )Vj ), are 
positive. Since the sign function in B is reduced there is an element b G B such 
that the elements u r (^b~ l ,bvj are positive in B for any u G S, v G T. Hence, 
Sc^ 1 C Pos(G) and fcc^T C Pos{G) } as required. 

Case 2. io < jo- ln this case the reduced form of each u G S 1 is of the type 
u = iti . . .M r ( u )C _1 , u r t u ) G S, and furthermore, u r ( u ) G Pos(B). Observe that 
for every v G T since ui> G Pos(G) then in the product c _1 w either c _1 cancels 
out completely or cr x v G Pos(G). In the former case i> = cu' and the reduced 
form of v' is of the type v[ . . . v' m ; in this event put y(v) = v[. In the latter 
case, put y(v) — 1. By construction the sets <Sb = {u r (u) « £ S} C J] and 
T = {y(u) | u G T} C S arc such that 5 B T B C Pos(B). Hence there is 6 G £> 
with C Pos(B),bT B C Pos(B). It follows that S'c6" 1 G Post(G) and 

6c _1 T C Pos(G) as claimed. 

□ 

Lemma 4.6. Le£ 6e non-trivial groups with strongly reduced sign func- 

tions and G = A * B equipped with the standard free product sign func- 
tion. If L G Rat{G) and for some elements u,v G G uLv C Pos(G) then 
uLv G Rat{Pos{G)). 

Proof. We use induction on c(L). If c(L) = then L is finite and the claim is 
obvious. Now consider the following cases. 

Case 1) If L = L\ U L 2 and c(Li) < c(L) for i = 1,2 the result follows by 
induction. 

Case 2) Suppose L = LiL 2 and c(L.j) < c(L) for i = 1,2. Then wLw 
is a product of two rational sets uL\ and L 2 v. Since the standard free prod- 
uct sign function on A * B is reduced there is an element w G G for which 
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uLiw -1 , V0L2V C Pos(G). Then uLiw^ 1 ,wL2V £ Rat(Pos(G)) by induction, 
hence uLv = uLiw -1 • wL-iv £ Rat(Pos(G)). 

Case 3) Suppose L = L*, where c^L^ < c(L). Then uiw = uL\v = 
uv(v~ l Liv)* , and u; = uv £ Pos{G) since 1 G L*. Denote L2 = v~ x L\v. 
so uLw = wL^ and 0(^2) = c(w _1 Liw) = c(L\) < c(L). 

If L2 C Pos{G) then the result follows by induction. Otherwise, there is 
an element Z £ L2 \ Pos(G). If Z = li...lik+i...lt is the reduced form of Z then 
there exists i such that Zj is negative and all the factors Zj+i, . . . ,lt are positive. 
Suppose that the reduce form of Z is written in the form 

Z = r i T 1 ...r^ 1 ui...u k r 1 ...r t , 

where either k = 1, or k > 1 and iti«fc 7^ 1. 

Claim 1. In the notation above the factor Zj of Z is among the first i + 1 
factors r^ 1 , . . . , rf i.e., i < t + 1. 

Indeed, the reduced form of every element d £ if-Z^ must have the product 
K-i— T a * the en dj otherwise the element dZ £ wij would not be positive. 

If i > |Z|/2 then for every d £ wL\ more then half of Z cancels out in dZ, so 
|dZ| < |d|. This implies that wL\ does not have an element of minimal length, 
i.e., wL* 2 = 0, which contradicts the fact that w £ wL^. 

Assume now that i < \l\/2, but i > t + 1. It follows that k > 2. In this 
case for any natural number p one has V £ an( i the rightmost occurrence 
of the negative factor Zj of Z does not cancel in l p (since it is in the core Z of 
I). Therefore, for sufficiently large p the rightmost occurrence of the factor Zj 
does not cancel in wl p , so the element wl p is negative, which contradicts the 
condition wh* 2 C Pos(G). Hence i < t + 1 and the claimed follows. 

Let's take Z £ L2 \ Pos(G) such that i = i(Z) be the maximal possible such 
index among all elements Z in L2 \ Pos(G) (such z exists since wlj £ Pos(G)). 
We can also assume that Zj = 6 £ £?. It follows that the reduced form of if is 
equal to w'b(w)l~} 1 ...Zr 1 , where b(w) £ £? and 6(10)6 £ Pos(B). 

Claim 2. The reduced form of any element d £ L2 can be written as d = 
d'&COC-V-Jr 1 . where 6(d) £ B and 6(d) 6 £ Pos(B). 

Indeed, suppose m £ L2 does not have Z"^..^ 1 at the end. Then m = 
mi...m s l~ 1 ...li , where < q < i — 1 (we assume m = m\...m s for q = 0) 
and m s l q+ i 7^ 1 (notice that m s and Z^+i are in the same factor). Since wml 
is positive the negative factor Zj of Z cancels out in wml, so TOi...m s _i(m s Z g +i) 
must cancel out in wra. The element w ends on If s > q then m = 

li...l q m q+ i...m s l~ 1 ...l^ 1 , and m q +i = Z g +i. On the other hand m s l q +i 7^ 1, 
which shows that m q+ i...m s is the core of m. If s 7^ g + 1 then the length of 
the core of m is greater than 1, so for sufficiently large integer p the element 
wm p l £ wL*2 is negative - contradiction. If s = q+1 then m = l\...l q l q+ il q 1 ...l^ 1 

and wml = w'Z ( ^ 1 Z g+ iZ 9+ iZ ( j + 2 ■ ■ ■ Zt, whose reduced form is w'l q +il q +2 ■ ■ - h, so 
it contains Zj - contradiction. The case s < q can be done similarly. This proves 
the claim. 

In the notation above, since the sign function on B is strongly reduced there 
is a positive element bo £ B for which bob, 6(w)6^~ 1 and 6(d)6p 1 are positive 
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for all d G L 2 . Denote r = b$l i \...l 1 . Notice, that r is positive. It follows 
from Claim 2 that every element d € L 2 can be written in the reduced form as 
d = d'(b(d)bo 1 )r. Let 

L 3 = {rf'(W 1 ) \deL 2 } 

Claim 3. The language rL 3 is positive. 
Indeed, it follows from the argument above. 

Notice that rL 3 = rL 2 r~ x so rL 3 is rational and c(rL 3 ) = c(L 2 ) < c(L). 
Hence by induction rL 3 G Rat{Pos(G)). Now 

wL\ = w'r{L 2 }* = w'{rL 3 }*r, (3) 

where all factors in product on the right are positive. Hence uLv = wL^ G 
Rat(Pos(G)), which proves the lemma. 

□ 

Lemma 4.7. Let A, B be non-trivial groups with strongly reduced sign functions 
and G = A * B equipped with the standard sign function. If L £ Rat(G) and 
L C Pos(G) then L e Rat(Pos(G)). 

Proof. We use induction on complexity c{L) of L. If c{L) = then L is finite 
and the claim is obvious. Consider the following cases. 

Case 1) If L = L\ U L 2 or L = L\ with c(L,) < c(L), i = 1,2, then Li,L 2 C 
Pos(G) and the result follows by induction. 

Case 2) Suppose L = L\L 2 . By Lemma 14.51 the standard free prod- 
uct sign function on G is reduced. Therefore there is u G G such that 
L x u,u~ x L 2 C Pos(G). By Lemma U u. v,- l L 2 C Rat(Pos(G)). Hence 
L = (Liu)(u~ 1 )L 2 C Rat(Pos(G)), as claimed. 

□ 

Lemma 4.8. Let A, B be two non-trivial groups with strongly reduced sign func- 
tions and such that the sets Rat(A), Rat(B) are closed under intersections and 
complements (form Boolean algebras) and G = A* B . If the submonoid Pos(G) 
relative to the standard sign function on G is rational then for any L G Rat(G) 
the intersection L = L n Pos(G) is rational in the monoid Pos(G). 

Proof. Observe, that under the premises of the theorem Rat(G) is a Boolean 
algebra by G.A. Bazhenova's result [I], which states that class of groups with 
Boolean algebras of rational subsets is closed under free products. Hence L is 
rational in G. Then L G Rat{Pos{G)) by Lemma Wl\ 

□ 

Corollary 4.9. Let F 2 = F{X 2 ) be a free non-abelian group of rank 2 with 
basis X 2 = {xi,x 2 }. Denote by X% the free submonoid of F 2 generated by X 2 . 
If L G Rat(F 2 ) then L = LDX*e Rat(X*). 

Proof. Follows from Lemma l4~8l 

□ 
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5 Free groups and free products 

Let F = F(X) be a free non-abelian group with basis X = {xi, X2, ■■■}■ Put 
X% = {xi,x%} and consider the free group F^ = F(X2) with basis X2 as the 
distinguished subgroup of F(X) generated by X^. By we denote the free 
submonoid of F2 generated by X2 . 

Lemma 5.1. Let G = A * B be a free product of two non-trivial groups A and 
B. If w is a proper word such that the verbal subgroup w(G) has infinite index 
in G then the set w[G] is not rational in G. 

Proof. Observe that for such w the verbal subgroup w(G) is a non-trivial normal 
subgroup of infinite index in G. By B. Baumslag's result [3] the subgroup w(G) 
is not finitely generated. Since a subgroup generated in a group by a rational 
subset has to be finitely generated (see [8], Theorem 4.2), the generating set 
w[G] of w(G) is not rational in G. □ 

Corollary 5.2. Let G = A * B be a free product of non-trivial groups A 
and B with infinite abelianization G a b = G/[G,G]. Then for any non-trivial 
commutator word w the set w[G] is not rational in G. 

Corollary 5.3. Let F be a free non-abelian group. Then for any non-trivial 
commutator word w the set w[F] is not rational in F. 

For the rest of the paper we fix a proper non-commutator word w. Observe, 
that e = e(w) > 2. To apply Rhcmtulla's criterion we view the free group F2 
as a free product F2 = (xi) * (X2) = A * B, equipped with the standard sign 
function given by the submonoid X%. This sign function is strongly reduced. 

Lemma 5.4. Letp,q € X| and E C X| be such thatpE*q C w[F 2 }. Then one 
of the following hold: 

1) \u\ > 2 for every u G E* . In this case supp{pE*q) is finite. 

2) \u\ = 1 for every u £ E* . In this case either E* C x\ or E* C x\. 

Proof. Let u, v £ E* . Since E C X| the elements u, v are positive, so they are 
equal to their cores u = u,v = v. Assume that u, v are given in the reduced 
forms 

u = u 1 u 2 .-.u k , v = v 1 v 2 --vi. 

We prove first that if k, I > 2 then supp(u°) = supp(v°). Notice, that either 
supp(u Q ) = {ui,U2, Uk} for even k, or supp(u a ) = {U2, U3, (itfeiti)} for odd 
k. 

Since 

puu{v}*uuq C X2 fl tyfi^] (4) 

the Rhemtulla's criterion shows that supp(u ) C supp(v°). Indeed, \v\ > 2 
implies that the length of v n strictly grows with n, so if b is a factor in supp(u°) 
but not in supp(v°) then for infinitely many k there is a number n = n(k) such 
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that the word puuv n uuq contains precisely one 6-gap of length k. So the gap 
function 5b.k(u) is equal to 1 for infinitely many k on puu{v}*uuq. Notice that 
S b -i k (puuv n uuq) = since the words are positive. Hence the function jb,e is 
unbounded on w[-F2] - contradicting the Remtulla's criterion. Similarly we show 
that supp(v°) C supp(u°). Hence supp(u ) = supp(v ) as claimed. 

Observe now that if there are elements u,v £ E such that \u\ > 2 and \v\ = 1 
then for sufficiently large n one has \u\, \uv n \ > 2 and supp(u°) ^ supp((uv n )°) 
- contradicting the statement above. 

The argument above shows that either all elements in E* arc of syllabic 
length greater then 1, or all of them have length 1. This proves 2) and the first 
part of 1). To finish the proof observe that for any u = u\ . . . Uk £ E* one 
has U2, ■ • ■ , Uk-i, UkUi £ supp(v°). Since u±, Uk, U\Uk are positive there are only 
finitely many choices for u\ and Uk as divisors of UkU\. This proves that there 
is a finite set K C X^ such that for any u £ E* supp(u) C K. It follows that 
supp{pE* q) is finite as claimed. 

□ 

Corollary 5.5. Let L G i?ai(X|) Ru;^]- Then there is a finite set K L C x\\Jx2 
and a natural number n = n(L) such that every element u £ L can be presented 
as a product of the following type: 

U = S\tiS2 ■ ■ ■ S n t n , 

where supp(si) C Kl and U EijUij. 

Proof. Since L £ i?at(X|) it can be presented in the form 

L = cinE^ . . . CLijE^aij+i . . . E^.aui+i 

For Eij as above, put E = Eij and denote p = an . . . aij,q = oty+i . . . au^+i € 
X% . Since each En contain 1 we have pE*q C Rat^X^) H ^[i^]- Now the result 
follows from Lemma [5T4l □ 

Theorem 5.6. Let F be a free non-abelian group and w be a proper word. Then 
the set w[F] is not rational in F. 

Proof. Let w be a proper word such that w[F] is rational in F. Notice that 
e = e(w) > 2 since w is proper. By Corollary [53] the word w is not a commutator 
word. Observe that L = w[F] n F2 = w[F2] is rational in F2 as a homomorphic 
image of a rational set under standard homomorphism F — > F2. By Corollary 
14.91 the set L = L n X| is rational in X^ . By Corollary 15.51 there is a finite set 
Kl C ij U and a natural number n = n(L) such that every element u G L 
can be presented as a product of the following type: 

u = si*is 2 . . . s n t n , (5) 

where supp(si) C i^^ and G U x^. Chose ( G N large enough so a;^ ^ Kl. 
Chose I £ N such that I > Then the word u = {x\x2) le belongs to 

w[F2], hence it belongs to L. However, u cannot be presented in the form ([5]) - 
contradiction, which proves the theorem. 

□ 
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Theorem 15.61 can be generalized into free products as follows. 

Theorem 5.7. Let A and B be groups containing elements of infinite order 
X\ G A, X2 G B, and G — A * B. If the rational sets Rat(A) and Rat(B) are 
Boolean algebras then for every proper word w with e(w) > 2 the set w[G] is not 
rational in G. 

Proof. Notice first that by Bazhenova's result [1] the set Rat(G) is a Boolean 
algebra. Obviously, the subgroup generated by x\ and xi in G is a free subgroup 
i<2 with basis {x\,x-z\. If w[G] G Rat(G) then L = w[G) n is rational in G 
by P] , hence by another Bazhenova's result [2] L is rational in F2 . By Corollary 
14.91 the set L — L (~l X% is rational in X| = {xi, X2}* and so has a presentation 
of the form (flj in X%. Since u^i 7 ^] C u>[G] and the free decompositions of F2 
is induced from the free decomposition of G one can complete the proof by an 
argument similar to the one from the proof of Theorem l5.6l 

□ 

Corollary 5.8. Let A and B be groups containing elements of infinite order, 
and G = A * B. Let the abelianization G a b is infinite, and the rational sets 
Rat(A) and Rat(B) are Boolean algebras. Then for every proper word w the 
set w[G] is not rational in G. 

Proof. Follows from Corollary [5721 and Theorem l5~7l □ 

Corollary 5.9. Let A and B be infinite finitely generated abelian groups and 
G = A* B. Then for every proper word w the set w[G] is not rational in G. 

Proof. Bazhcnova showed in [5] that rational sets in finitely generated abelian 
groups form Boolean algebras. Now the result follows from 15.81 □ 

Theorem 15.71 and Corollaries 15.81 and 15.91 have far reaching generalizations. 
To explain we need the following simple but useful result. 

Lemma 5.10. Suppose that a group H admits a homomorphism onto a group 
G in which every set w[G] for a proper word w is not rational. Then for every 
proper word w the set w[H] is not rational in H. 

Proof. Suppose that (p is a homomorphism of H onto G. Since for every word 
w we have w[G] = <p(w[H]), and a homomorphic image of any rational set is 
rational (see (5]) w[H] G Rat(H) implies that w[G] G Rat(G) that contradicts 
our assumption. Hence, w[H] Rat(H). 

□ 

Corollary 5.11. Suppose that a group H admits a homomorphism onto a free 
non-abelian group F. Then for every proper word w the set w[H] is not rational 
in H. 

There are many classes of groups which have free non-abelian quotients. We 
list some of them below. 
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Corollary 5.12. In the following groups H for every proper word w the set 
w[H] is not rational: 

1) Pure braid groups PB n for n > 3. 

2) Non-abelian right angled Artin groups. 

3) Finitely generated non-abelian residually free groups. 

Proof. To prove 1) observe that a pure braid group PB n ,n > 3, has the group 
PB3 as its epimorphic quotient (see [3], for example), and the group PB3 is 
isomorphic to F2 x Z, so PB n , n > 3, has the free group F2 as its quotient. 

To see 2) Let G = G(T) be a non-abelian partially commutative group 
corresponding to a finite graph F. Then there are three vertices in F, say 
V\ 1 «2 j ^3 such that the complete subgraph Tq of T generated by these vertices 
is not a triangle. In particular, a partially commutative group Go = G(To) is 
either a free group F3 (no edges in Tq), or (Z x Z) * Z (only one edge in Tq), 
or F2 x Z (precisely two edges in To). Notice that in all three cases the group 
G(Tq) has F2 as its epimorphic quotient. Now, it suffices to show that G(Tq) is 
an epimorphic quotient of G(T), which is obtained from G(T) by adding to the 
standard presentation of G(T) all the relations of the type v = 1, where v is a 
vertex of T different from vi, 1)2,1)3. This shows that F2 is a quotient of G(T) 

By definition every non-abelian residually free group has a free non-abelian 
quotient, so 3) holds. □ 

The following result shows that a wide class of free products of groups does 
not have rational proper verbal subsets. 

Corollary 5.13. Suppose that H = C * D, where the factors C, D are finitely 
generated groups with infinite abelianizations. Then for every proper word w 
the set w[H] is not rational in H. 
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